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Introduction 


Philadelphia’s Out-of-School Time (OST) system has identified K-34 grade literacy support as the focus of 
the first phase of its strategic plan, launched in 2017. Research shows that OST programs can have an 
impact on early literacy skills,! and this plan views OST programs as an important “delivery system” for 
addressing critical needs of youth throughout the city, and aims to better coordinate and leverage 
resources for OST programs to address these needs.” Over the next few years, Philadelphia’s OST system 
will undertake a focused effort to improve reading outcomes for K-3"4 grade participants. 


To inform its early literacy work, the City OST system asked Research for Action (RFA) to conduct 
two pieces of research: 1) a literature review of evidence-based programs for delivering effective 
OST literacy support; and 2) a scan of Philadelphia’s OST literacy programs and practices as 
compared to these evidence-based programs. The literature review is summarized in the report 
Supporting Literacy in Out-of-School Time: A Review of the Literature, released in July 2017.3 The report 
identified evidence-based programs and the Key Ingredients of high-quality OST literacy programs. The 
report also includes a tool which enables OST providers to assess whether they have these Key Ingredients 
in place. 


This report assesses the degree to which Philadelphia’s OST literacy programs have the Key 
Ingredients necessary for high-quality literacy programming. It also identifies areas that require more 
support. Findings are based on interviews with two leaders in Philadelphia’s early literacy landscape, staff 
from six OST literacy programs, and a survey completed by staff from 48 OST programs. 


+ Yael Kidron and Jim Lindsay. "The Effects of Increased Learning Time on Student Academic and Nonacademic Outcomes: Findings from a Meta- 
Analytic Review. REL 2014-015." Regional Educational Laboratory Appalachia. (2014).; Zakia Redd, Christopher Boccanfuso, Karen Walker, Daniel 
Princiotta, Dylan Knewstub and Kristin Moore. “Expanding time for learning both inside and outside the classroom: A review of the evidence base.” 
Child Trends. (2012).; Stanley T. Crawford. “Meta-analysis of the impact of after-school programs on students reading and mathematics 
performance.” Doctoral dissertation, University of North Texas. (2011).; Patricia A. Lauer, Motoko Akiba, Stephanie B. Wilkerson, Helen S. Apthorp, 
David Snow and Mya L. Martin-Glenn. “Out-of-school-time programs: A meta-analysis of effects for at-risk students.” Review of Educational Research. 
(2006). 

2 City of Philadelphia, “Philadelphia OST Operational Plan” (2017). 

3 https://8rri53pmO0cs22jk3vvqna1ub-wpengine.netdna-ssl.com/wp-content/uploads/2017/09/RFA-Supporting-Literacy-in-OST-Summary-of- 
Evidence-June-2017.pdf 


Key Findings 


Description of OST Literacy Programs in our Sample 
e The vast majority of OST literacy programs in our sample provided year-round literacy 
support in multiple ways. Providers reported offering both afterschool and summer literacy- 
focused enrichment programs and also reported infusing literacy into other program activities. 
About one-third of survey respondents also provided tutoring and about half offered computer- 
based literacy support. 


e OST literacy programs in our sample targeted a range of literacy outcomes, but some 
interviewees expressed uncertainty regarding the appropriate literacy outcomes for OST. 


e OST literacy programs in our sample reported serving large proportions of struggling 
readers and English Language Learners (ELLs). Providers also reported that a lack of 
information about students’ special education status and/or their Individualized Education Plans 
(IEPs) made it difficult to ascertain the specific needs of some students. 


Presence of Key Ingredients for High-Quality OST Literacy Programs 
e High-quality program content: The vast majority of providers in our sample had many of the 
ingredients for high-quality program content in place. However, providers could improve in 
providing content designed by literacy experts, using school data for diagnostic and progress 
monitoring purposes, and having books available in languages other than English. 
o Structured Content: Do OST programs intentionally structure content through utilizing a 
daily protocol, curriculum, or lesson plans? 
= Strength: Most OST literacy programs were structured, primarily through the use of 
daily protocols that guide the way time is allocated during literacy programming. 
= Gap: Fewer programs, particularly tutoring programs, structured content through 
lesson plans developed by a literacy expert or with a packaged curriculum. These 
strategies would provide robust literacy content that is informed by early literacy 
expertise. 
o Aligned Content: Do OST programs align content with the school day to enhance in-school 
learning? 
= Strength: More than half of all types of OST literacy programs in our study are 
aligning their program content to the Pennsylvania common core. The Pennsylvania 
common core standards are available online by grade level. In addition, they provide 
broad guidelines that may be easier for providers to address than school-specific 
information. 
= Gap: Less than half (20-47%) of OST literacy programs align content to the school 
curricula or foci as communicated by the school. These school specific strategies for 
alignment may be more difficult for providers to adopt because they require greater 
coordination with schools. 
"Gap: Some programs (20-31%) had no alignment strategies. Alignment was a 
greater challenge for enrichment programs compared to tutoring programs. 
o Access to Content: Do OST programs provide participants access to engaging and relevant 
books? 
= Strength: All OST programs provided access to books at appropriate reading levels. 


= Strength: OST programs provide access to books that reflect different cultural and 
ethnic groups, are perceived by providers to be high interest and available for 
children to take home. 

=" Gap: Fewer programs provide access to books in different languages. 

o Informed Content: Do OST programs inform their program content with data about 

student needs? 
= Strength: A majority of providers have access to some source of diagnostic and 
progress monitoring data, primarily their own assessments and conversations with 
teachers. 
=" Gap: About half of providers report not having access to school data, the source that 
may be the most reliable indicator of a child’s reading ability. 
Highly-qualified staff: Far fewer providers had the ingredients for high-quality staffing. 
Particular gaps were noted with regard to use of certified teachers and/or providing sufficient pre- 
service training for paraprofessional staff. However, close to half were accessing literacy expertise 
through POSTLI which may have filled these gaps for some providers. 
o Credentialed Staff: Do certified teachers provide literacy instruction? 
=" Gap: OST providers reported challenges with hiring certified teachers. 
« Potential strength: A majority of frontline literacy staff have some college education. 

o Prepared Staff: Pre-Service Training: In lieu of certified teachers, do staff and volunteers 
receive sufficient training? 

= Strength: The vast majority of tutoring programs (92%) were on track for providing 
an adequate amount of pre-service training. 

= Gap: Less than half of enrichment programs were on track for providing an 
adequate amount of pre-service training. 

o Prepared Staff: Ongoing Support: In lieu of certified teachers, do staff and volunteers 
receive sufficient support? 

= Strength: The majority of providers offered monthly observation and feedback. 
=" Gap: While coaching was offered, few providers had an internal or external 
(POSTLI) literacy expert providing coaching. 

o Literacy Expertise: Do OST programs have a literacy expert—a reading specialist or 
someone certified to teach early literacy—delivering or overseeing programming and 
staff training? 

= Strength: More than two-thirds of providers responding to our survey had access to 
literacy expertise, either through staff with literacy expertise or through an external 
support (i.e., POSTLI). 
=" Gap: One-third of providers did not have access to literacy expertise (i.e., a reading 
specialist or someone certified in early literacy). 
Sufficient literacy instruction and varied program activities: Varied activities to support 
engagement were common, but finding time for literacy activities was a challenge. Most 
afterschool programs offered sufficient hours of literacy instruction while most summer programs 
did not. 

o Hours of Literacy Programming: Do OST programs provide weekly and yearly hours of 

programming sufficient to have an impact? 
= Strength: More than two-thirds of afterschool enrichment programs met both 
weekly and yearly thresholds for hours of literacy instruction. 


= Gap: Less than one-third of summer enrichment programs met the weekly targets 
for hours of instruction but slightly more (36%) met the yearly targets. 
"Gap: Nearly half of providers (44%), including both afterschool and summer, 
reported that finding time for literacy activities was at least a slight challenge. 
o Varied Program Activities: Do OST programs provide fun and engaging activities that 
differentiate the program from the regular school day? 
= Strength: Most OST providers reported strategies to differentiate the program from 
the regular school day. 
e ELL gaps: Philadelphia OST literacy programs struggled to meet the needs of ELLs. Most 
providers did not have access to books in other languages or provide training on how to support 
ELL students. Providers also reported language barriers in engaging ELL parents. 


Conditions to Support Literacy Programming 
e Providers reported that obtaining adequate funding, access to school data, and sufficient 
parent involvement were challenges faced by their literacy programs. Research suggests these 
conditions support high-quality programming. 
e Providers identified other key local resources for supporting their programs including 
volunteers from colleges and universities and partnerships with other community organizations to 
support engaging activities and provide literacy expertise. 


Recommendations 
This research points to the following recommendations to build on the strengths and close the gaps in OST 
literacy efforts in Philadelphia: 


Recommendations for the OST System 
Facilitate a system-wide discussion about how OST can support early literacy. 


e Encourage the use of high-quality, evidence-based programs. 

e Encourage tutoring. 

e Allow for variation in programs and practices adopted. 

e Foster other practices to support literacy that draw on the strengths of OST programs. 


Encourage collection and use of data. 


e Support OST providers with data collection and outcomes assessment. 
e Provide guidance for OST providers regarding the early literacy skills that most need targeting for 
different grade levels and subgroups of students. 


Fill gaps in training and literacy expertise. 


e Expand literacy-focused training opportunities. 
e Provide training and other supports for OST providers with regard to ELL students. 
e Provide access to literacy expertise for program development, training, and coaching. 


Engage key partners: universities and funders. 


e Engage colleges and universities as sources of literacy expertise, as well as volunteers. 
e Engage funders in considering system-wide strategies for supporting OST literacy programs. 


Recommendations for Future Research 

This scan of OST literacy practices, along with RFA’s July 2017 literature review, illuminated some aspects 
of OST literacy programming, but this research has limitations and points to important gaps that should be 
addressed by future research. These include: 


e ©The efficacy of non-traditional programs for OST literacy support; 

e The role of drop-in programs in supporting early literacy; 

e The most effective strategies for supporting early literacy of ELL students in OST settings; 

e The relationship between Key Ingredients and outcomes; 

e The evidence base for afterschool enrichment programs and programs using paraprofessional staff; 
and 

e The role of external literacy experts, particularly POSTLI, in boosting the quality of OST literacy 
programs. 


In addition, as new early literacy efforts are launched, research should examine the following: 


e The quality of implementation; and 
e The role of the OST system in supporting early literacy efforts. 
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|. Introduction: The Importance of Fostering Early Literacy in Out 
of School Time Programs in Philadelphia 


In 2017, about three-quarters of Philadelphia’s third and fourth grade students did not read at grade level.4 
Yet failure to achieve grade-level reading proficiency by fourth grade is a precursor to significant 
educational challenges, including drop out.5 While teaching children to read and supporting their literacy 
development is a task that falls largely to schools and parents, segments of the broader community can also 
play a role. In particular, research shows that Out-of-School Time (OST) programs can have an impact on 
early literacy skills.¢ 


Philadelphia’s OST system has identified K-3"4 grade literacy support as the focus of the first phase of its 
strategic plan, launched in 2017. This plan views OST programs as an important “delivery system” for 
addressing critical needs of children and youth throughout the city and aims to better coordinate and 
leverage resources for OST programs to address these needs.’ Over the next few years, Philadelphia’s OST 
system will undertake a focused effort, the parameters of which are still being determined, to improve 
reading outcomes for K-34 grade participants. 


The City’s OST literacy efforts will build on work done by Philadelphia’s Read by 4th Campaign, which seeks 
to engage the broader community in supporting all Philadelphia schools with their early literacy goals. 
Read By 4 leadership has already engaged OST providers as partners, and has invited them to serve on 
committees to identify strategies and priorities for boosting literacy. The OST literacy efforts will also build 
on work begun by the Philadelphia Out-of-School Time Literacy Initiative (POSTLI) in 2000. POSTLI serves 
as a technical assistance provider for OST literacy programs and sponsors an intensive literacy program, 
Youth Education for Tomorrow (YET), which is being implemented by fourteen OST providers in 
Philadelphia. 


479% of third grade students and 72% of fourth grade students, School District of Philadelphia website. 

5 Donald J. Hernandez. “Double jeopardy: How third-grade reading skills and poverty influence high school graduation.” Annie E. Casey Foundation 
(2011). 

6 Yael Kidron and Jim Lindsay. "The Effects of Increased Learning Time on Student Academic and Nonacademic Outcomes: Findings from a Meta- 
Analytic Review. REL 2014-015." Regional Educational Laboratory Appalachia. (2014).; Zakia Redd, Christopher Boccanfuso, Karen Walker, Daniel 
Princiotta, Dylan Knewstub and Kristin Moore. “Expanding time for learning both inside and outside the classroom: A review of the evidence base.” 
Child Trends. (2012).; Stanley T. Crawford. “Meta-analysis of the impact of after-school programs on students reading and mathematics 
performance.” Doctoral dissertation, University of North Texas. (2011).; Patricia A. Lauer, Motoko Akiba, Stephanie B. Wilkerson, Helen S. Apthorp, 
David Snow and Mya L. Martin-Glenn. “Out-of-school-time programs: A meta-analysis of effects for at-risk students.” Review of Educational Research. 
(2006). 

’ City of Philadelphia, “Philadelphia OST Operational Plan” (2017). 


To inform its early literacy work, the City OST system asked Research for Action (RFA) to conduct 
two pieces of research: 1) a literature review of evidence-based programs for delivering effective 
OST literacy support; and 2) a scan of Philadelphia’s OST literacy programs and practices as 
compared to these evidence-based programs. The literature review is summarized in the report 
Supporting Literacy in Out-of-School Time: A Review of the Literature, released in July 2017.8 The report 
identified evidence-based programs and also the Key Ingredients of high-quality OST literacy programs. 
The report also includes a tool which enables OST providers to assess whether they have these Key 
Ingredients in place. 


This report assesses the degree to which Philadelphia’s OST literacy programs have the Key 
Ingredients necessary for high-quality literacy programming. It also identifies areas that require more 
support. Findings and definitions from RFA’s literature review inform this report and are described in 
Table 1. 


Table 1. Findings from the Literature Review of OST Literacy Programming 
18 High-Quality OST Literacy 
Programs: Programs with 
strong or moderate evidence of 
impact, or based on strong 
theory but needing more 


Conditions for High- 
Key Ingredients of High-Quality OST Literacy Programming: | Quality OST Literacy 


Common practices observed across evidence-based programs Programming: External 
supports that enable 


research program success 

Four types of programs High-quality program content Conditions 

e One-on-one tutoring e Structured: Program intentionally structures content e Adequate funding 

e Afterschool enrichment by utilizing a curriculum, daily protocol, or lesson plans. | e School 

e Summer enrichment e Aligned: Program content aligns with school day to partnerships 

e Non-traditional including enhance in-school learning. e = Parental 
computer-based and e Access: Participants have access to engaging and involvement 
literacy-infused, i.e., relevant books. e Infrastructure to 


programs that do not 


h fic f e Informed: Content is informed by data about student support the use 
gins . aie piewciaaiiaias needs. of volunteers 
literacy but infuse 
literacy into other Highly-qualified staff 
activities such as arts = ae 
or sports e Credentialed: Certified teachers provide literacy 

instruction. 


e Prepared: When not hiring certified teachers, staff and 
volunteers are given sufficient training and support. 

e Literacy expertise: Programs have a literacy expert 
delivering or overseeing programming and staff 
training. A literacy expert is a reading specialist or 
someone certified to teach early literacy. 


Sufficient literacy programming and varied program 
activities 


e Hours of literacy programming: Sufficient weekly and 
yearly hours of programming to have an impact 

e Varied program activities: Fun and engaging 
activities that differentiate the program from the 
regular school day 


8 https://8rri53pmO0cs22jk3vvqna1ub-wpengine.netdna-ssl.com/wp-content/uploads/2017/09/RFA-Supporting-Literacy-in-OST-Summary-of- 
Evidence-June-2017.pdf 


Guiding Research Questions 


Building on the results of our literature review of high-quality OST literacy practices, this report addresses 
the following research questions: 


e What are the primary OST programs for K-34 grade literacy support in Philadelphia, who do they 
serve, and what outcomes do they assess? 

e How do Philadelphia’s OST literacy programs vary in their alignment to the Key Ingredients of 
evidence-based programs identified in the literature review and what are the gaps and limitations? 

e What conditions exist to support OST literacy programming and what barriers or challenges do 
providers encounter? 


e What more is needed to address challenges or fill gaps in OST literacy programming in 
Philadelphia? 


Methodology and Sample 


To address the research questions, RFA conducted interviews with and a survey of Philadelphia’s 
OST providers. In both, we explored the practices of local OST literacy programming as well as the 
conditions supporting or impeding these programs. Table 2 describes the group of providers that 
participated in each of these activities. 


Table 2. Survey and Interview Sample 


[on 


| 48 providers of K-3rd grade literacy responded to 
the survey.? This includes: 
e 29 respondents from a list of 81 
identified literacy providers?° 
19 additional providers from a broader Six providers identified by the City OST 
list of 266 OST providers1*+ system as providing K-3rd grade OST 
literacy programming; two leaders in 
42 providers reported offering more than one Philadelphia’s early literacy landscape. 
type of literacy programming including after- 
school and summer programming. (See Table 3 
on pg. 7 for more details.) 


Affiliation 
with Read by 69% of respondents (29 respondents*) reported __ Five of six providers were affiliated with 


Ath or affiliation with either Read by 4th or POSTLI. Read by 4th and/or POSTLI. 
POSTLI 


77% (37 respondents) were non-profit 


organizations. : : ; 
g Five providers were non-profit 


organizations and one was a City 


Organization = The remainder includes schools/universities, City aeuaiinent 


departments, religious institutions and other 
organizations. 


52% of afterschool enrichment programs (22 
respondents) were funded by DHS Four of six interviewed providers 


DHS received some funding from DHS for 


Funding** 44% of summer enrichment programs (18 afterschool or summer programs. 
respondents) were funded by DHS 


9 63 providers opened the survey but only 48 answered at least one question and provide K-3 literacy programming. 

10 A list of 81 OST literacy providers in Philadelphia was developed by PhillyBOOST, Read by 4th and POSTLI. We had a 35% response rate from this 
list. OST providers on this list received an invitation followed by four reminder emails over the two week period of survey administration (June 14*- 
June 30*). Read by 4 and POSTLI also issued reminders to their networks to complete the survey. 

11 PhillyBoost also distributed the survey to their broader list of OST programs including arts, sports and general youth development programs. The 
invitation for this group and the first page of the survey emphasized that this survey was meant only for programs with a significant literacy 
component to their programming, in order to screen out programs without those components. From this broader group we had a 7% response rate. 
Two reminder emails were sent to this group in addition to the initial request to complete the survey. 

12 Respondents were not required to answer every question on the survey. Therefore, sample sizes vary by question asked. Throughout this report, we 
provide the number of respondents that selected each individual survey item. 

13 RFA only queried providers about DHS funding because it is the top funding source for OST programs in the City and because RFA’s previous report 
found that providers receiving this funding reported more quality assurance practices. 


Generalizability of Sample 

Little data exists on the larger population of OST providers in Philadelphia. Therefore, it is difficult to 
determine the extent to which this survey sample is representative and generalizable to the larger 
population. However, the variety of providers that answered the survey suggests that we captured a 
broad range of OST literacy programs operating in Philadelphia. As shown above, the survey sample 
differs along two key dimensions -involvement in formal partnerships about OST literacy through 
Read by 46 and POSTLI (two-thirds were involved in these partnerships while one-third were not); 
and receipt of funding from Philadelphia’s Department of Human Services, the largest funding stream 
for OST programs in Philadelphia (about half receive this funding while about half do not). RFA’s 
report Scanning the System, released in July 2017, found that programs receiving DHS funding were 
more frequently engaged in quality improvement practices and had more stable staffing than other 
groups of programs. Therefore, the survey sample includes providers that are more likely to have 
strong OST literacy programs as well as those that are less likely to have strong programs. While this 
variation is important, we were not able to examine outcomes for these subgroups due to the small 
sample sizes. 


Key Findings 
Description of OST Literacy Programs in our Sample 
e The vast majority of OST literacy programs in our sample provided year-round literacy 
support in multiple ways. Providers reported offering both afterschool and summer literacy- 
focused enrichment programs and also reported infusing literacy into other program activities. 
About one-third of survey respondents also provided tutoring and about half offered computer- 
based literacy support. 


e OST literacy programs in our sample targeted a range of literacy outcomes, but some 
interviewees expressed uncertainty regarding the appropriate literacy outcomes for OST. 


e OST literacy programs in our sample reported serving large proportions of struggling 
readers and English Language Learners (ELLs). Providers also reported that a lack of 
information about students’ special education status or their Individualized Education Plans (IEPs) 
made it difficult to ascertain the specific needs of some students. 


Presence of Key Ingredients for High-Quality OST Literacy Programs 

e High-quality program content: The vast majority of providers in our sample had many of the 
ingredients for high-quality program content in place. However, providers could improve in 
providing content designed by literacy experts, using school data for diagnostic and progress 
monitoring purposes, and providing books in languages other than English. 

e Highly-qualified staff: Far fewer providers had the ingredients for high-quality staffing. 
Particular gaps were noted with regard to use of certified teachers and/or providing sufficient pre- 
service training for paraprofessional staff. However, close to half were accessing literacy expertise 
through POSTLI which may have filled these gaps for some providers. 

e §©Sufficient literacy instruction and varied program activities: Varied activities to support 
engagement were common, but finding time for literacy activities was a challenge. Most 
afterschool programs offered sufficient hours of literacy instruction while most summer programs 
did not. 


ELL gaps: Philadelphia OST literacy programs struggled to meet the needs of ELLs. Most 
providers did not have access to books in other languages or provide training on how to support 
ELL students. Providers also reported language barriers in engaging ELL parents. 


Conditions to Support Literacy Programming 


Providers reported that obtaining adequate funding, access to school data, and sufficient 
parent involvement were challenges faced by their literacy programs. Research suggests these 
conditions support high-quality programming. 

Providers identified other key local resources for supporting their programs including 
volunteers from colleges and universities and partnerships with other community organizations to 
provide literacy expertise. 


Report Overview 
This report is organized in the following sections: 


1. 


Description of OST Literacy Programs in our Sample: In this section, we describe OST programs 
responding to our survey by type, outcomes targeted and assessed, and literacy needs addressed. 
Presence of Key Ingredients of High-Quality OST Literacy Programs: In this section, we assess 
whether and how characteristics of Philadelphia’s OST literacy programs align with the Key 
Ingredients of high-quality OST programs. 

Conditions to Support High-Quality OST Literacy Programming: In this section, we compare 
how important conditions in Philadelphia’s local environment compare to the conditions present in 
the environment of high-quality programs. 

Conclusions and Recommendations: Finally, we offer recommendations based on this research 
for Philadelphia’s OST system and OST providers. 


Description of OST Literacy Programs in our Sample 


This section describes the OST literacy support that was provided by programs in our sample. We report on 
the prevalence of each program type (tutoring, afterschool enrichment, summer enrichment, and non- 
traditional), the outcomes they targeted and assessed, and the literacy needs of participants served. 


A. Type of OST Literacy Support 


Almost all of the surveyed providers (88%) offered multiple types of OST literacy support. The vast 
majority (83%) reported offering afterschool and summer enrichment programs, and three-quarters 
(77%) also offered literacy-infused activities, i.e., activities that integrate literacy into arts, sports or other 
recreational activities. Table 3 summarizes these results. 


Table 3. Percentage and Number of Providers that Offer Each Type of Literacy Programming 


Percentage Number of 
of providers providers 
(%) C2) 
One-on-one literacy tutoring 35% 17 
Afterschool literacy enrichment+4 88% 42 
Summer literacy enrichment 85% 41 
Both afterschool and summer enrichment 83% 40 


Non-traditional 


e Computer-based:15 Programs offering literacy-focused e 48% e 23 
computer programs. 
e __Literacy-infused: Programs integrating literacy practices e 77% e 637 


into other activities (e.g., sports, drama)- either after 
school or in the summer 


Offer more than one type of literacy programming 88% 42 


Because there was insufficient evidence about non-traditional programs (computer-based and literacy- 
infused activities) in the literature review, we do not have quality indicators for these type of programs. 
Therefore, we do not assess the alignment of Philadelphia’s non-traditional literacy programs to Key 
Ingredients of high-quality programs. 


B. Outcomes Targeted and Assessed 
OST providers reported targeting multiple literacy outcomes. Nearly all (97%) focused on at least one 
of the five skill areas identified by the National Reading Panel (NRP)1¢ and 54% targeted more than one. 
Table 4 below displays the percent of providers targeting each of these outcomes in their programs. 


Table 4: Outcomes Targeted by Providers 


NRP skill areas Other literacy outcomes 


f F General 7 
Phonics ATU Fluency | Vocab peas fi Reading SOILS Writing 
Awareness Comprehension : Attitudes 
achievement 
Afterschool | % | 47% 53% 74% 87% 90% 50% 76% 71% 
aoe # | 18 20 28 33 34 19 29 a7 
Summer % | 54% 62% 81% 81% 95% 32% 81% 76% 
programs # | 20 23 30 30 35 12 30 28 


14 Seven of the afterschool enrichment programs were YET centers, one of the 18 evidence-based programs identified in RFA’s July 2017 literature 
review. YET originated in Philadelphia and is based on a balanced literacy program. This program was identified in the literature review as having 
strong theory but limited evidence with more research required. It is implemented with support from the Philadelphia Out-of-School Time Literacy 
Initiative. 

15 Only a handful of computer-based programs used one of the three evidence-based computer programs identified in the literature review: Fast 
ForWord, DaisyQuest, and Earobics. Other programs being used by providers include: Lexia, Reading Eggs, and iReady. 

16 The National Reading Panel recommends reading instruction built on five main, interconnected components: phonemic awareness, phonics, 
fluency, vocabulary, and comprehension. 


Table 4 shows: 

e Fluency, vocabulary, and reading comprehension were the most commonly targeted NRP skill 
areas. Almost all afterschool and summer programs targeted reading comprehension while three- 
quarters or more targeted vocabulary and fluency. 


e About three-quarters of all programs reported targeting reading attitudes and writing. This may 
have been in addition to or instead of the five skill areas. Two providers expressed more confidence in 
OST’s ability to influence reading attitudes as compared to specific literacy skills. One provider 
explained: 


I think what afterschool can do on its own to actually increase reading levels is not small, but it’s not 
test scores either. We only have them for a short period of time. I think where we can best support kids 
is making them want to learn how to read, making them interested in reading. 


Similarly, a leader in the Philadelphia early literacy landscape asserted that OST providers should focus on 
supporting enjoyment and positive attitudes towards reading and writing rather than on building specific 
literacy skills. Studies have shown a strong relationship between positive reading attitudes and reading 
achievement.!7 Much of the discussion and research about supporting literacy development in OST 
programs focuses on reading, but targeting writing skills can help to support broad literacy learning, 
including reading. 


OST providers reported using measures of general reading achievement to assess student progress. 
These measures include grades in classes and reading levels on report cards, which do not provide 
feedback on the specific skill areas that providers reported targeting in their programming. 

Table 5 below displays the assessments used by providers to measure these outcomes. 


Table 5: Assessments Used by Providers'® 


Formal Literacy Assessments General Reading Achievement 


Reading Informal No 


DIBELS | AlIMSweb | PSSA | DRA | Grades | level on reading Assessment 
report card | inventory Used 
Afterschool | % 3% 3% 12% | 24% | 59% 50% 26% 3% 
SS re 1 4 8 20 17 9 1 
Summer % 0% 3% 6% 18% | 47% 41% 41% 3% 
n+ «60 ft 2 6 16 14 14 1 


Table 5 shows: 

e The most commonly used assessments were grades, reading levels on student report cards, 
and an informal reading inventory, all of which assess general reading achievement. Overall, 85 - 
88% of providers assess general reading achievement though only one-third to half targeted 
general reading achievement. This suggests providers were assessing a broader outcome than they 
were actually targeting with their programs. 

e Few providers used other forms of standardized or diagnostic assessments such as Dynamic 
Indicators of Basic Early Literacy Skills (DIBELS), Academic Improvement Measurement System 


17 De Naeghel, Jessie, Hilde Van Keer, Maarten Vansteenkiste, and Yves Rosseel. "The relation between elementary students' recreational and 
academic reading motivation, reading frequency, engagement, and comprehension: A self-determination theory perspective." Journal of Educational 
Psychology 104, no. 4 (2012): 1006.; Martinez, Rebecca S., O. Tolga Aricak, and Jeremy Jewell. "Influence of reading attitude on reading 
achievement: A test of the temporal-interaction model." Psychology in the Schools 45, no. 10 (2008): 1010-1023. 

18 These questions were not asked of tutoring providers. 


(AIMSweb), Pennsylvania System of School Assessment (PSSA), or Developmental Reading 
Assessments (DRA).!9 


C. Literacy Needs of Philadelphia OST Program Participants 


Providers reported serving a large proportion of struggling readers and ELL students. More than 
three-quarters of providers perceived half or more of their participants to be struggling readers. Almost 
50% of providers reported their populations to be at least half ELL students. This percentage is high 
considering that about 10% of students in the School District of Philadelphia are ELL students.2° Providers 
serving ELL students may be overrepresented in our survey sample or providers maybe unaware of the 
formal ELL classification of their participants. (See Appendix A the proportions of struggling readers and 
ELLs by program type.) 


In spite of the prominence of struggling readers, not all OST providers have access to information about 
students’ special education status or their Individualized Education Programs (IEPs). As a result, they may 
not be aware of the specific needs of their students or school personnel with whom they should partner 
(i.e., reading specialists or special education teachers). As one provider explained: 


IEPs are not something that the school shares with us. We ask parents. Parents don’t always share. 
Sometimes they do, but it’s something that we’re working on with the City to get better inclusion into 
that process. Because if we’re working with the kids afterschool, we want to do it to the best of our 
ability. That includes knowing what’s in their IEP. 


Consistent with the literature review, tutoring programs were most likely to report targeting literacy 
supports to only struggling readers. In contrast, most of the afterschool and summer enrichment OST 
programs offered literacy supports to every participating student. (See Appendix A for percentages of 
programs by type that target their literacy supports.) 


Ill. Presence of Key Ingredients of High-Quality OST Literacy 


Programs 


This section of the report examines the extent to which providers responding to our survey have 
incorporated the Key Ingredients of high-quality OST literacy programs into their OST efforts (see Table 1). 
Key Ingredients fall into three categories: (1) high-quality program content, (2) highly-qualified staff, and 
(3) sufficient literacy programming and varied program activities. The degree to which each Key Ingredient 
is present in a particular program varies. If providers reported that a Key Ingredient was at least minimally 
present, it was considered “on track.” If a Key Ingredient was not present at all, it was considered “off 
track.” Below, we begin with an overall assessment of the degree to which Key Ingredients were present in 
our sample, and follow with a more detailed analysis of provider strengths and gaps. 


A. Overall Assessment of Key Ingredients 


Table 6 displays the eleven Key Ingredients of high-quality OST literacy programs, and presents the percent 
and number of providers which reported having these elements in place. 


19 Reading levels on report cards may come from DRAs administered by school staff. Both report card reading levels and DRA’s were given as 
assessment options in the survey, in order to differentiate between DRAs administered by OST staff as opposed to teachers. 

20 Lin, J., Hughes, R., Long, D., Kim, D. (2016). Characteristics of English Language Learners in the School District of Philadelphia. Philadelphia 
Education Research Consortium, Philadelphia, PA. http://3I59p62inuOt2sj11uLhh23l-wpengine.netdna-ssl.com/wp- 
content/uploads/2016/06/PERC-ELL-16-06-28-Student-Characteristics-Brief_Final.pdf 


Table 6. Percentage and Number of OST Literacy Providers On Track with Key Ingredients of High-Quality OST 
Literacy Programming by Program Type2+ 


Key: Red text = Less than half of providers were on track 

Afterschool Summer enrichment 
enrichment programs on track 
programs on track with Key Ingredient 
with Key Ingredient 


Deas ea esa es 
Program Content 


Structured: Have a curriculum or 67% i 40 69% | 24 is) ad 
structured program 
Aligned: Have content aligned to school 80% i 49 7a 5 69% 24 
day (e.g., curriculum / standards) 


Tutoring programs 
on track with Key 


Ingredients for high-quality OST literacy 


programming Ingredient 


Access: Have access to books oe 16 85% 33 Gee 36 
Informed: Have access to diagnostic data 13% | a 81% | 30 74% 26 


Informed: Have access to progress 92% : 412 95% | 35 100% | 30 
monitoring data 
Staffing Practices 


Credentialed: Staffing — at least half 15%22 22 9% 3 9% 3 
certified teachers 
Prepared: Staff receive adequate pre- 92%23 11 48% 14 39% 13 
service training 


Prepared: Staff receive ongoing (at least 73% 11 77% 27 83% 30 
monthly) support 
Literacy Expertise: Staff has access to 69% 9 68% 25 67% 24 
literacy expertise (internally or externally24) 
Sufficient Literacy Instruction and Varied Program Activities 


Ez : 

(Summer | 

rayon a 
ae 68% 24 26% 8 

) i : 

(Afterschool — . 

tutoring) 


100% 15 91% 38 90% 37 


Table 6 reveals the following notable findings: 


Across program types, more providers are on track for providing high-quality program content 
than highly-qualified staffing or hours of instruction. 


21 Each provider was asked about these ingredients for each type of program that they conduct. Because many providers have more than one type of 
program and their practices might vary among their programs, we were not able to compute an ‘overall’ column in this table. 

22 Evidence-based tutoring programs did not use certified teachers therefore, this is not a quality indicator for tutoring programs. However, we include 
the percentage of tutoring programs hiring certified teachers for comparison. 

23 The pre-service target for tutoring programs was much lower (at least 1 hour) than other program types (10 or more hours). 

24 This includes all providers who reported receiving some services from POSTLI, including YET centers and providers receiving coaching and 
professional development. Overall, 18 survey respondents reported receiving supports from POSTLI. 

25 Tutoring practices were the same in the summer and the school year across all areas except potential dosage where the number of hours offered 
differed. Therefore, we report both summer and school year tutoring programs in this cell. 

26 Tutoring programs have an inherent non-academic component—positive adult youth relationships. However, research has not explored the ways in 
which the quality of tutor-tutee relationships impact the success of tutoring. 
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e The majority of providers (67-100%) are meeting the minimum threshold for providing high-quality 
program content. More details about these standards and provider program content practices are 
provided in Section B.1 below. 

e Fewer than half of providers are on track with most of the highly-qualified staffing ingredients. More 
details about these standards and provider staffing practices are described in Section B.2 below. 

e Almost all providers offered activities distinct from the school day. Yet less than one-third of summer 
programs and one half to two-thirds of afterschool programs met the minimum threshold for hours of 
literacy instruction. More details about hours of instruction and varied activities are provided in 
Section B.3 below. 


B. Taking a Closer Look at Key Ingredients of High-Quality OST Literacy 


Programming 


This section provides more details about provider practices with regard to the three high-quality 
programming practices: program content, staffing, and sufficient literacy instruction and varied program 
activities. We examine the strengths and gaps in each of these Key Ingredient areas. 


1. High-Quality Program Content 
A. Structured Content 
Overall assessment: More than 65% of 


A. Structured: The program intentionally structures providers offered structured program content. 
content through utilizing a curriculum, daily protocol, (See Table 6). 


Key Ingredients of High-Quality Program Content 


or lesson plans. 
Aligned: Program content aligns with school day to 
enhance in-school learning. 


We asked providers what strategies they used to 
plan and structure literacy programming. At 
minimum, providers needed to have either a 
curriculum, structured protocol, or lesson plans 
developed by a literacy expert to be considered 
on track. While all these strategies provide 
structure, they vary in the extent to which they 
provide robust literacy content. Table 7 displays 
the percentages of survey respondents who report using an array of approaches to structuring the content 
of their programs. Respondents were able to select multiple options to these survey questions, so the 
values in Table 7 do not total to 100%. 


. Access: Participants have access to engaging and 
relevant books. 

. Informed: Content is informed by data about 
student needs. 
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Table 7. Approach to Instructional Planning Used by Survey Respondents 


On Track instructional planning strategies 


Use lesson plans from Develop own lesson 
Use structured 2 ‘1 
internally developed or plans (by literacy 
protocols 
packaged curriculum expert) 
% # % # % # 
Tutoring 53% 8 ae | 20% 2 
programs 
Afterschool 
enrichment 57% 20 31% 11 18% 6 
programs | 
Summer 
enrichment 61% 22 39% 14 19% 6 
programs 


Table 7 identifies the following strengths and areas for improvement in structuring program content. 


Strength: Most OST literacy programs were structured, primarily through the use of protocols that 
guide the way time is allocated during literacy programming. 


Gap: Fewer programs, particularly tutoring programs, structured content through lesson plans 
developed by a literacy expert or with a packaged curriculum. These strategies would provide robust 
literacy content that is informed by early literacy expertise. Among providers using a packaged curriculum, 
only a handful reported using one of the curricula found in high-quality programs: KidzLit, Read for 
Success, Reading Coaches, YET, and Book Buddies. See Appendix B for additional information on the 
curricula providers used. 


Interviews revealed that some providers based decisions about curricula on their suitability given the 
staffing and time constraints of the OST setting. One provider reported concerns about using an existing 
literacy curriculum because they perceived these curricula to be as too prescriptive and difficult to adapt to 
the OST environment. However, another provider chose to use a packaged curriculum because staff lacked 
literacy expertise. Specifically, this provider chose KidzLit because it is built for an afterschool time frame, 
is accessible to OST staff, provides activities and instructions, and has socioemotional components. 


B. Aligned Content 
Overall assessment: More than 65% of providers aligned program content with in-school content (see Table 


6). 


Providers were asked if the content of their literacy programs aligned to school day content. Alignment 
could be achieved via aligning to the school curriculum or to the common core state standards, or providers 
could receive other information from the school (e.g., daily lessons or assignments) to align OST 
programming. In order to be considered on track in this area, providers had to indicate using at least one of 
these approaches. Table 8 below displays providers’ school alignment strategies. 
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Table 8. Providers that Use Strategies for Aligning Program Content to Enhance In-School Learning 


School provides 


Align with Align with PA : 3 None of the 
F information to 
school curricula common core ‘ above 
align2? 
% # % # % # % # 
Tutoring programs 47% | 7 | 67% 10 20% 3 | 20% 3 
eee CULE 31% 41 57% 20 37% 13 | 29% 10 
Sa wae ge 20% 7 51% 18 34% 12 31% 11 


Table 8 suggests both strengths and gaps in alignment of OST program content to enhance in-school 
learning. 


Strength: More than half of all types of OST literacy programs are aligning their program content to 
the Pennsylvania common core. The Pennsylvania common core standards are available online by grade 
level. In addition, they provide broad guidelines that may be easier for providers to address than school- 
specific information. While alignment to the PA Common Core is a helpful first step for program alignment, 
strategies that are specific to school curricula and classroom activities might be even more effective 
because they create a more seamless learning experience for OST participants. 


Gap: Less than half (20-47%) of OST literacy programs align content to the school curricula or foci 
as communicated by the school. These school specific strategies for alignment may be more difficult for 
providers to adopt because they require greater coordination with schools. To align to a school’s curricula, 
OST providers need to learn from the school what curricula they use and, if not available online, obtain 
copies of the curricula from the schools. To align with daily classroom activities, providers would need to 
build relationships with classroom teachers; a difficult task since most OST staff are not in the school 
building during the school day. 


Gap: Some programs (20-31%) had no alignment strategies. Alignment was a greater challenge for 
enrichment programs compared to tutoring programs. Because tutoring programs tend to serve 
struggling readers and can provide individualized support, these programs may seek greater alignment to 
the school curriculum. 


C. Access to Content 
Overall assessment: More than 80% of providers reported giving participants’ access to books (see Table 6). 


Providers were asked if they used books in literacy activities, and were queried about the types of books 
they used. Access to books was the minimum standard needed for providers to be considered on-track in 
this area. However, appropriate reading levels, cultural relevance, and the ability for students to take books 
home were also deemed important in the literature. In addition, providing books in different languages is 
important for supporting ELLs. Table 9 below displays the percentage of providers reporting literature- 
based best practices with regard to book access. 


27 The survey question asked providers to indicate if the school provided information to plan programming in line with school day activities. These 
activities could include information on daily lesson plans, grade level themes, or school-wide initiatives. 
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Table 9. Percentage and Number of Providers That Report Best Practices for Book Access 


Participants can access books that... 


% # 
Are at an appropriate reading level 100% j 33 
Are about different cultural and ethnic groups 91% j 30 
Are engaging 91% 30 
Can be taken home 70% 23 
Are in different languages 42% 14 


Table 9 indicates strengths and gaps with regard to this Key Ingredient. 


Strength: All OST programs provided access to books at appropriate reading levels. However, one 
provider voiced a rationale for not emphasizing leveled books, saying “We're still very much about freedom 
of choice...and not prescribing what kids should or shouldn't read—you decide what you want to learn.” An 
interview with a leader in the Philadelphia early literacy landscape reconciled both attitudes toward 
reading levels, explaining that OST providers should help children find books that are of interest to them 
and can be read independently, while maintaining a focus on enjoyment of reading and writing. 


Strength: Most OST programs provided access to books that reflect different cultural and ethnic 
groups, are perceived by providers to be high interest for participants, and are available for 
children to take home. 


Gap: Less than half of programs provide access to books in different languages. 


D. Informed Content 
Overall assessment: More than 70% of providers reported having access to diagnostic and progress 
monitoring data (Table 6). 


Providers were asked if they had access to data for diagnostic and progress monitoring purposes and, if so, 
the type of data available. High-quality programs typically had access to this data which enabled them to 
provide targeted literacy supports. This data allowed providers to assess the skills with which their 
students needed most help and to adjust their supports as students progressed over the course of the year. 
Data types included school data, providers’ own assessments, and information from conversations with 
teachers. Providers had to indicate access to at least one type of data to be considered on track.28 Table 10 
below displays strategies for accessing diagnostic and progress monitoring data. The first column shows 
what percentage of providers have access to any form of diagnostic or progress monitoring data. The 
remaining four columns show how they access that data, and providers were allowed to select multiple 
options. 


28 The survey did not ask whether providers had this data for all students or some subset of students. 


14 


Table 10. Strategies for Accessing Diagnostic and Progress Monitoring Data 


Percentage of providers that access data by... 


Have access = Tajking to Looking at setae ak Another 
toanydata teachers school data method 
assessments22 
% # % # % # % # % # 


, Tutoring programs Tome 11 (ee s ee 7 60% § 9 oe 2 
~ H : : : : 
3 Afterschool 81% 30 51% 19 46% 17 46% 17 8% 3 
D enrichment programs 
a Summer enrichment TA% 26 | 34% 12 | 40% : 14 49% 17 11% 4 
programs 
= Tutoring programs 92% 12 |46% 6 |62% > 8 62% 8 
ne 
SEEN Afterschool 95% § 35 | 51% 19 |49% 18 | 62% | 23 
° [= enrichment programs 
“Se- Summer enrichment | 100% 30 | 43% 13 |53% 16 | 63% © 19 
programs 


The following strengths and gaps with regard to this Key Ingredient are suggested in the table above. 


Strength: A majority of providers have access to some source of diagnostic and progress monitoring 
data, primarily their own assessments and conversations with teachers. Tutoring programs were 
more likely than afterschool and summer programs to administer their own diagnostic assessments. All 
types of programs were equally likely to administer their own progress monitoring assessments during 
programming. 


Gap: About half of providers report not having access to school data, the source that may be the 
most reliable indicator of a child’s reading ability. A comparable percentage of providers accessed data 
through conversations with teachers and administering their own assessments. However, it is unclear 
whether these conversations with teachers reveal the type of information providers need to adequately 
target literacy supports. For example, providers that lack reliable diagnostic data might have difficulty 
understanding whether books are at the appropriate reading level for individual children in their 
programs. 


Summary of High-Quality Program Content Ingredients 

OST providers responding to our survey were on track in many of the program content areas; however, 
gaps remain. Most notably, while providers had structured program content, many were not using 
packaged curricula or lesson plans designed by literacy experts. In addition, while providers were able 
to provide access to books, only a third had books in other languages for ELL students. Finally, providers 
were engaged in diagnostic or progress monitoring but most did not have access to school data. 


29 The survey did not query the type of diagnostic and progress monitoring assessments that were used when a provider specified this response. 
However, the assessments that providers reported using for measuring student outcomes were reported in Section Il. Some of these assessments 
would be appropriate for and were likely used for diagnostic and progress monitoring. 
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2. Highly-Qualified Staff 
A. Credentialed Staff 
Overall assessment: Across program types, less 


A. Credentialed: Certified teachers provide literacy than 20% of providers reported staffing literacy 
instruction. programs with certified teachers (see Table 6). 


Key Ingredients of Highly-Qualified Staff 


B. Prepared: In lieu of certified teachers, staff and 
volunteers are given sufficient training and 
support. 

Literacy expertise: Programs have a literacy 
expert delivering or overseeing programming and 
staff training. A literacy expert is a reading 
specialist or someone certified to teach early 
literacy. 


Providers were asked about the qualifications of 
the staff who delivered literacy instruction. To be 
considered on track with this ingredient, 
afterschool and summer enrichment programs 
had to indicate that certified teachers delivered 
literacy instruction. Since many programs did not 
meet that target, it is informative to understand 
the qualifications of frontline staff who are not 
certified teachers. Table 11 displays the 
qualifications of frontline literacy staff. 


Table 11. Percentage of Providers that Report Half or More of Frontline Staff Have Each Education Level 

Some college through High school, 

Bachelor’s degree diploma, or GED 
% # % # 


62% 15% 


Teaching certificate 


% 


Tutoring programs 15% 


# 
2 
Afterschool enrichment programs 3 
Summer enrichment programs 3 


The following strengths and gaps in frontline staffing are suggested in Table 11. 


Gap: OST providers reported challenges with hiring certified teachers. Few providers hired certified 
teachers as frontline staff. Interviewed providers noted difficulties meeting the salary requirements of 
teachers, and also reported that many teachers were unwilling to commit to afterschool positions. As one 
afterschool provider explained, “As much as we try and get teachers, it’s hard to afford it on an afterschool 
budget. It’s also part-time work. Teachers work all day and then getting them to commit afterschool for an 
entire school year...it’s a big ask of a teacher. “ 


Two interviewed providers utilized certified teachers in their enrichment programs. Both recruited 
teachers from the schools in which they offered programs, and both noted the added cost of hiring certified 
teachers. One described teachers as the key driver of the program’s cost while the other provider simply 
described the program as expensive. 


Potential strength: A majority of frontline literacy staff have some college education. Research on 
high-quality OST programs has found that higher levels of staff education are associated with better 
academic outcomes.?° Therefore, it is encouraging that at least half the staff at a majority of OST literacy 
programs have at least some college education. 


30 Denise Huang and Ronald Dietel. "Making afterschool programs better (CRESST Policy Brief)." Los Angeles, CA: University of California (2011); Beth 
M Miller, "Pathways to success for youth: What counts in after-school." (2005); Jenell Holstead and Mindy Hightower King. "High-quality 21st Century 
Community Learning Centers: Academic achievement among frequent participants and non-participants." Journal of Education for Students Placed at 
Risk JJESPAR) 16, no. 4 (2011): 255-274. 
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Similar to the finding reported in RFA’s Scanning the System report, providers, in both interviews and 
surveys, cited “recruiting and retaining high-quality staff” as one of their top challenges. In Scanning the 
System, 60% of providers reported frontline staff with less than one year of experience in their 
organization, and less than 30% reported that the majority of their frontline staff had been in their role for 
more than 5 years. On the OST literacy survey, 32% of providers identified staff turnover as one of their top 
three challenges. 


B. Prepared Staff: Pre-Service Training 

Overall assessment: More than 90% of tutoring programs were on track with pre-service training while less 
than half of academic and summer enrichment programs were on track. However, pre-service training 
requirements varied by type of OST program (see Table 6). 


Providers were asked about the amount and topic of preservice training provided to staff. Providers were 
assessed as on track if they met the standards for the amount of pre-service training offered by program 
type. Standards varied by type. Tutoring has the lowest standard for the amount of pre-service training 
(one hour), while academic and summer enrichment had higher standards (ten hours). Table 12 below 
displays the percentage of providers that met the pre-service training threshold for that program type. 


Table 12. Amount of Pre-service Training for Frontline Staff 


Amount of pre-service training for frontline staff 


On track 
No pre-service Less than 5 + ne as 10 or more Ms a ea aor 
training hours hours Se aneunenu alas) 
10 hours 
% # % # 
; 92% gu 
Tutoring programs 25% i 
(minimum threshold = 1 hour) 


Afterschool enrichment 
programs 


45% | 13 

(minimum threshold = 10 hours) 
Summer enrichment 36% 12 
programs 


(minimum threshold = 10 hours) 


Table 12 suggests the following strengths and gaps in pre-service training. 


Strength: The vast majority of tutoring programs (92%) provided at least one hour of training,31 the 
minimum threshold for tutoring programs to be on-track in this area. In four of five evidence-based 
tutoring programs, tutoring also received ongoing support from literacy content experts or certified 
teachers after tutoring began. More information about training topics covered by providers is included in 
Appendix C. 


Gap: Less than half of enrichment programs were providing 10 or more hours of pre-service 
training, the minimum threshold for enrichment programs to be on-track in this area. To understand 
the barriers that might prevent providers from conducting enough training, respondents were asked to 
identify the aspects of training that present a challenge. Table 13 displays their responses and shows that 
providers listed cost and time as the most common barriers to providing enough training for their staff. 


31|f providers reported that they offered training for less than five hours, we assumed they provided training for at least an hour. The smallest 
category on the survey was less than 5 hours. 
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Table 13. What Factors Make Staff Training Challenging? 


Major/moderate factor 


% # 
Cost 88% eal 
Time 80% 20 
Capacity 33% 8 
Expertise 32% 8 


C. Prepared Staff: Ongoing Support 
Overall assessment: More than 70% of providers offered ongoing training and support (see Table 6). 


Providers were asked about the type of ongoing training and support they offered, and how frequently they 
offered it. Providers had to be offering at least one of these supports at least monthly to be considered on 
track. However, the value of these approaches could vary based on the expertise of the staff member 
conducting them. Table 14 displays the percentage of providers that reported offering these different types 
of supports. 


Table 14. Percentage and Number of OST Literacy Providers That Offer Ongoing Supports 


External support 
Internal support offered at least monthly (teas een 


Observation and Non-expert se expert BOSTLI provides 
feedback?2 pious ecules coaching 
coaching coaching 
% # % # % # % # 

Tutoring programs 50% 7 | 2 
Afterschool enrichment 71% 25 9 
programs 
Summer enrichment 73% 28 410 


programs 


The table above identifies strengths and gaps in ongoing support. 


Strength: The majority of providers offered monthly observation and feedback. A higher percentage 
of afterschool and summer enrichment programs than tutoring programs conducted program observations 
and provided feedback to frontline staff at least once per month. 


Gap: While coaching was offered, few providers had an internal or external (POSTLI) literacy expert 
providing coaching. Only one-third of providers had literacy experts provide coaching at least monthly. At 
the same time, around one-quarter of providers reported receiving external literacy coaching supports 
from POSTLI. Our data do not allow us to determine how frequently POSTLI literacy coaching occurred. 


D. Literacy Expertise 
Overall assessment: Two-thirds of providers have access to a literacy expert, either on staff or from an 
external source (see Table 6). 


Providers were deemed to have the Key Ingredient of literacy expertise if a staff member was either a 
reading specialist or a teacher certified in early literacy. Providers could also meet this standard if they 


32 The survey did not ask who provided the observation and feedback. 
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worked with POSTLI, which could be considered an external source of literacy expertise. Figure 1 displays 
the percentage of survey respondents that had access to internal or external literacy experts. 


Figure 1. Percentage of OST Literacy Providers with Access to Literacy Experts 
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The figure above reveals the following strengths and gaps in literacy expertise among OST providers. 


Strength: More than two-thirds of providers responding to our survey had access to literacy 
expertise, either through staff with literacy expertise or through an external support (i.e., POSTLI). 
While most high-quality programs had literacy expertise on staff, a few accessed expertise from the 
organization that created their literacy model. In Philadelphia, literacy expertise is available through 
POSTLI. Around half of providers (44-62%) reported receiving POSTLI services; however, we lack 
information about the frequency and duration of these supports. Some interviewed providers purchased 
literacy models and/or curricula developed by external organizations. They had access to these developers 
to obtain pre-service and ongoing training and occasional monitoring of programs. 


Gap: One-third of providers did not have access to literacy expertise (i.e., a reading specialist or 
someone certified in early literacy). More than two-thirds of providers reported having staff they 
considered literacy experts but who did not meet this definition. Importantly, staff with strong literacy 
expertise were more likely to be deeply engaged in coaching, training and supervision than those who did 
not meet our literacy expert criteria. Similarly, the two of six supervisors we interviewed with literacy 
expertise focused on coaching and training staff, while the remaining four supervisors focused on logistics. 
Survey respondents reported that about half of literacy experts, including those meeting our definition and 
those not meeting it, were involved in developing the curriculum and lesson plans. 


Summary of Highly-Qualified Staff Ingredients 

The Key Ingredient of Highly Qualified Staff was not consistently present in programs participating in 
our study. Providers needed support in hiring and retaining qualified staff for frontline literacy work 
including certified teachers as well as staff with some higher education. In addition, they needed support 
in providing more hours of pre-service training. While a majority were providing some form of ongoing 
training and support, particularly observation and feedback, almost half did not have ongoing support 
from a literacy coach. In addition, one-third of respondents did not have access to either internal or 
external literacy expertise. 
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3. Sufficient Literacy Programming and Varied Program Activities 


Key Ingredients of Sufficient Literacy A. Hours of Literacy Programming 
Programming and Varied Program Activities: Overall assessment: More afterschool programs 
than summer programs met the threshold for hours of 


A. Hours of literacy programming: Sufficient 
literacy instruction (Table 6). 


weekly and yearly hours of programming to have an 
impact. Below is the minimum time required by Providers were asked how much time students spent 
program type: on literacy activities in their enrichment or tutoring 
© Tutoring: 1-2.5 hours per week; 30-60 programs per week and per year. Thresholds for 
hours per year minimum levels of weekly and yearly hours of 
Afterschool enrichment: 2 hours per literacy programming varied by program type. To be 
week; 44 hours per year assessed as on track, providers had to meet both the 
Summer enrichment: 6 hours per week; weekly and yearly thresholds. Figure 2 below shows 
44 hours per year the percentage of providers that met the weekly or 
yearly thresholds for hours of instruction. 


B. Varied program activities: Fun and engaging 
activities that differentiate the program from the 
regular school day 


Figure 2. Percent of Providers that Met Weekly and Yearly Threshold for Hours of Literacy Instruction 
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Figure 2 suggests both strengths and gaps in hours of literacy instruction. 


Strength: More than two-thirds of afterschool enrichment programs met both weekly and yearly 
thresholds for hours of literacy instruction. Two-thirds of afterschool tutoring programs met the 
threshold for weekly hours of instruction, but fewer (56%) met the yearly threshold. 


Gap: Less than one-third of summer enrichment programs met the weekly targets for hours of 
instruction but slightly more (36%) met the yearly targets. Almost all summer tutoring programs were 
meeting weekly targets for hours of instruction, but only 27% were offered for enough weeks to meet the 
yearly target for summer program hours of instruction. 
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On the survey, providers were asked if finding time for literacy activities was a slight, moderate, or 
significant challenge. 


Gap: Nearly half of providers (44%), including both afterschool and summer, reported that finding 
time for literacy activities was at least a slight challenge. About a quarter reported that it was a 
moderate or significant challenge. 


B. Varied Program Activities 
Overall assessment: 90-100% of providers reported that they offered varied program activities to engage 
participants (Table 6). 


Providers were asked whether program activities were varied and engaging. To be considered on track, 
providers only needed to offer one non-instructional activity. They reported a range of such activities, as 
noted below. 


Strength: Most OST providers reported strategies to differentiate the program from the regular 
school day. Three-quarters of respondents offered activities that fostered positive adult-youth 
relationships. Three-quarters also reported offering games (e.g., Scrabble). All surveyed afterschool and 
summer enrichment programs reported goals that went beyond literacy, such as fostering math and 
science skills, extracurricular skills, school engagement, or social emotional growth and development. 


In interviews, providers emphasized the importance of this non-academic element of programming, 
expressing a desire to not make OST feel “too much like school”. As one provider explained, “I think if you 
make afterschool [programs] just [resemble] the school day [but] longer, you lose a lot of the benefits.” 
Another provider explained that they seek to ensure that their program “is creative and engaging, so that 
we won't lose [students’] focus or attention in those afterschool hours.” One early literacy leader identified 
these non-academic components of OST as a strength that should be encouraged to ensure that literacy- 
rich OST environments are enjoyable and engaging. 


Summary of Sufficient Literacy Programming and Varied Program Activities Ingredients 
Summer programs had the most difficulty meeting standards for weekly and yearly hours of literacy 
instruction. Afterschool tutoring programs frequently met weekly but not yearly thresholds, while 
afterschool enrichment providers most often met both thresholds. Providers of all types reported a 
range of other program goals and activities that differentiate them from the school day. However, this 
variation made it difficult to provide sufficient time for literacy programming. 


In addition to the Key Ingredients discussed above, programs serving English Language Learners must 


consider additional promising practices in order to best serve ELL students. Below we highlight how 
OST literacy programs are addressing the needs of this population. 
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Spotlight: Are OST Programs Meeting the Literacy Needs of English 


Language Learners? 

The majority of OST programs responding to our survey served a high percentage of ELL 
students. Yet most did not have the supports in place to effectively serve this group. 
While research specifically focused on this topic is rare and provides little guidance on how to 
define high-quality programming, RFA included several questions in its survey addressing 
promising practices of quality OST literacy programming which were applicable to serving 
ELLs. Results are as follows: 


a. Connection and collaboration between home and program: About half of survey 
respondents (46%) reported language barriers as a major challenge for engaging 
parents. One provider also reported that translation for parent events is a significant 
added cost for programs that are already financially stretched. 

b. An inclusive environment respectful of home language and culture: Of the 39 OST 
programs that reported serving at least a few ELLs, 64% reported having books about 
different cultural/ethnic groups. Yet only 28% had books in languages other than 
English. 

c. Staff training: Less than one-third of survey respondents reported providing staff 
training on supporting ELL students. 


Other promising practices identified in the literature include the use of small groups, clear and 
explicit literacy instruction, and opportunities to practice speaking in low-risk, inclusive 
environments. 


IV. Conditions to Support High-Quality Literacy Programming 
High-quality OST literacy programming requires adequate funding, school partnerships, parent 
involvement, and infrastructure to support volunteer recruitment and retention. RFA asked providers 
about the presence of these supports in Philadelphia, and where they experienced barriers. In addition, we 
explored other conditions or supports that Philadelphia providers relied on to implement their programs. 
In this section, we report on providers’ experiences with each of the supporting conditions. 


A. Adequate Funding 


Surveyed providers were not asked about costs of their program, but they were asked the extent to which 
overall funding and adequate literacy materials were a challenge. 


The lack of adequate funding, including resources to purchase literacy materials, was among the 
top three barriers providers experienced. More than three-quarters of surveyed providers (82%) 
reported financial concerns to be a significant challenge. Interviewed providers described staffing as the 
biggest driver of cost, regardless of whether they used certified teachers or part-time staff. However, there 
was strong consensus that staffing is the most essential component of high-quality programming. Almost 
three-quarters of providers (72%) also reported that the cost of materials for literacy programming was a 
significant challenge. Costs were even higher for those programs supporting ELL students, who required 
materials in multiple languages and translators for parent events. 
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B. School Partnerships 


School partnerships are critically important to ensure that providers can align programs to the school day 
curriculum or core standards. Strong partnerships also help providers identify students who need extra 
support, obtain space in the school building for programming, and gain access to data on student learning 
needs and progress that could help OST providers deliver more targeted supports. 


Few surveyed providers (13%) reported school partnerships as a top challenge compared to other 
challenges they were facing. Yet more than a third (37%) reported that access to student data was 
one of their top three challenges. Providers reported difficulty in finding time to develop the partnership, 
finding the right school person to work with, and gaining the attention of school staff. 


Five of six interviewed providers--all large organizations serving students in multiple locations---described 
having positive relationships with schools. They may have had more capacity to cultivate relationships with 
schools/district staff than did smaller OST organizations. The Family Educational Rights and Privacy Act 
(FERPA) regulations may make obtaining individual student data directly from schools/districts difficult or 
impossible for many providers. One Philadelphia early literacy leader suggested that providers could ask 
parents to share students’ report cards, which contain reading levels and other information. Such requests 
could also provide a basis for sustained conversations with parents about their child’s reading level. 


C. Parent Involvement 
High-quality OST literacy programs cultivate parental support and involvement. Interviewed providers 
described their strategies for parent engagement, while surveyed providers were asked to report the 
extent to which parent engagement was a challenge for their programs. 


Only one provider described robust parent engagement strategies which placed parents at the center of 
their program, but their approach is instructive. This provider explained: “The only reason we’re able to 
generate dramatically different outcomes with the very same teachers and children that underperformed 
during the school year is by unlocking a world of one-on-one instructional time [with parents].” 


Strategies described by this provider included engaging parents in weekly workshops, during which they 
are trained to teach reading at home. In addition, all families participating in the program receive one home 
visit, which this provider explains “is a big part of the reason why we consistently get over 90% of families 
that attend the weekly training workshop.” An early literacy leader in Philadelphia reinforced the 
importance of parent engagement as a key support for early literacy efforts, and one which OST providers 
may be uniquely positioned to provide. 


While parental support is important, two-thirds of programs (68%) reported that engaging parents 
was somewhat or very challenging. Providers were asked to identify the factors that made parent 
engagement challenging. Results are displayed in Table 15. 


Table 15. What Is Challenging about Engaging Parents? 


Major /moderate factor 


% # 
Lack of staff time 91% 30 
Parents non-responsive 88% 29 
Language barrier 45% 15 
Not a priority of our program 13% 4 
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Table 15 shows: 


e Similar to school partnerships, engaging parents was made more challenging by lack of staff 
time and lack of parental response to outreach. Almost all reported these to be major or 
moderate factors in engaging parents. 

e Language barriers also created a challenge for almost half of providers. 

e However, few providers reported that parental engagement was not a priority for their 
program. 


D. Volunteer Infrastructure 
High-quality OST literacy programs that use volunteers have infrastructure in place to recruit, train and 
support these volunteers. Philadelphia’s OST providers were asked about the extent to which they used 
volunteers, and to describe their infrastructure for recruiting and supporting volunteers. 


About half of surveyed providers reported using volunteers in their programming. Between 46-53% 
of providers of all types of OST literacy programs used volunteers, alone or in addition to paid staff, to 
deliver literacy instruction to children. However, few programs relied solely on volunteers as frontline 
staff. 


Interviewed providers reported recruiting volunteers from colleges and universities. Two 
interviewed providers reported using volunteers but neither recruited them from any of the national 
service programs, a practice reported by high-quality programs. However, both recruited volunteers and 
work-study students from local colleges and universities, and these volunteers typically provided one-on- 
one reading support. One provider explained: 


They'll be helpful in just even reading with kids or listening to kids read or doing activities with 
students. I think one of the needs that we see the most that's not really educational is that kids just 
want someone to listen to them, someone to pay attention to them. They're just always so thrilled to do 
whatever as long as that adult will be one-on-one and focused on them. 


The other provider noted, “We have volunteers who are college students... and, if we want to do something 
like independent reading, we might pair them off with children to help with something like that.” RFA’s 
recent Scanning the System report also found that colleges and universities, in addition to word of mouth, 
were the most common source of volunteers for OST programs. 


Both interviewed providers that used volunteers were large organizations; one had extensive 
infrastructure to support volunteers and consequently had capacity to attract and support a large number 
of them. A staff member at one of these organizations described their infrastructure for supporting 
volunteers in this way: 


We're really lucky to have a volunteer services department here, and so we have a whole host of 
organizations that will come in and volunteer with us...We do have a website. I think they're using 
volunteer spots now to post needed activities and the volunteer coordinators will send up a job 
description for a volunteer. Then they'll search through and post it places and talk to their more 
regular volunteers and say, hey, do you know anyone that might—so it’s sort of a social media, word of 
mouth, website kind of a thing. 
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This provider also paid for volunteers’ clearances. 33 


The literacy survey did not ask providers about the amount of training provided for volunteers. However, 
RFA’s Scanning the System report found that 62% of OST providers who responded to a survey of quality 
assurance and staffing practices provided volunteer training. These providers averaged 2 training activities 
for volunteers per year for an average of 7 hours of yearly training. 


E. Other Community Partnerships 
While not identified as a key support in the literature review, all six of the OST providers interviewed 
described multiple partnerships with other organizations in the City that supported their OST 
literacy efforts. Partnerships were utilized for a variety of reasons including expanding program activities 
and literacy expertise. Two providers described activity partners such as museums or OST support 
organizations like the After School Activities Partnership (ASAP) that provided resources and ideas to help 
to make OST literacy activities more fun. These organizations provided games and other less traditional 
literacy activities such as Scrabble Clubs. One explained, “I think it’s definitely hard, but finding the 
partners to work with you is what makes a strong program. The best things that we do are bringing in the 
strong partners.” 


Three providers described partnerships which helped them access literacy expertise. These included 
partnerships with POSTLI, Read by 4", and other national organizations providing evidence-based 
programs and training to support these programs. As one provider stated, “I think, again, the partnerships 
we have bring in expertise that we'll never get in a staff that we can only pay from 3:00 to 6:00. Our staff 
are great, but bringing in partnerships lets us diversify the knowledge base that our staff has.” POSTLI is a 
unique strength of Philadelphia’s OST literacy environment. High-quality programs in our literature review 
did not have access to similar local resources for literacy expertise. 


Partnerships also supported parents’ literacy activities at home with children and for themselves. Two 
providers described partnerships that helped them with data collection and evaluation. One of these 
partnerships was with POSTLI, which offered tools for measuring outcomes and shared outcomes reports. 
The other was with PhillyBOOST, the City’s OST system, which has a database available to OST providers. 


All six providers were associated with Read by 4 and one became more focused on offering traditional 
literacy activities as a result. That provider explained, “We really more purposefully decided...that we 
needed to be more focused on helping kids learn to read in a more traditional way because for whatever 
reason, they weren't getting it at school or they missed it or something happened.” 


Summary of Supporting Conditions 

Providers reported several key conditions in place to support successful literacy programming. These 
included supportive school partnerships, the prevalence of colleges and universities through which to 
recruit volunteers or work-study students, and other community partnerships, including a unique 
resource for literacy expertise—POSTLI. 


33 Clearances are background checks legally required to work with children. In Pennsylvania, those who work with children are required to obtain a 
child abuse clearance, state criminal history record, and FBI criminal background check. 
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V. Conclusions and Recommendations 


This report examined the extent to which the characteristics of OST literacy programs represented in our 
research aligned to high-quality programs and Key Ingredients identified in RFA’s literature review. 
Providers reported offering year-round literacy supports using multiple strategies. They also reported the 
presence of Key Ingredients in a number of areas -particularly with regard to high-quality program 
content. Fewer programs reported the Key Ingredients of highly-qualified staffing and adequate hours of 
literacy programming. 


Providers served a diverse population of students but reported that the majority were struggling readers 
and almost all served at least a few ELL students. However, they lacked adequate information about the 
needs of special education students, and they lacked strategies and supports to address the needs of ELL 
students. As a group, providers described some divergence between the literacy outcomes they target and 
those they assess, i.e., targeting one or more of the NRP’s five skill areas, but most often assessing the 
broader outcome of general reading achievement. 


Gaps in OST literacy support were driven to some extent by limited resources for staffing, training, and 
materials; limited access to data for more targeted programming; and perhaps limited information about 
evidence-based programs and Key Ingredients for high-quality literacy programming in the OST setting. On 
the other hand, providers identified some key supports for literacy work in the City. These included 
support from the existing local literacy partners- Read by 4th and POSTLI- as well as national partners. 
With the exception of providing data, providers reported their school partners are generally supportive of 
their efforts. Finally, two providers identified colleges and universities as key resources for identifying 
volunteers or even part-time staff. 


As the City of Philadelphia develops strategies for the first phase of the OST strategic plan, this research 
points to the following recommendations to build on the strengths and close the gaps in OST literacy efforts 
in Philadelphia. 


A. Recommendations for the OST System 


1. Facilitate a system-wide discussion about how OST can support early literacy. 

e Encourage the use of high-quality, evidence-based programs. OST providers expressed some 
ambivalence about their ability to impact literacy skills. However, RFA’s literature review identified 
18 high-quality OST literacy programs that had strong or moderate evidence of effectiveness or a 
strong theory coupled with promising evidence of effectiveness. Few OST providers responding to 
our survey were using one of these programs. For example, while half of providers offered some 
form of computer-based literacy support, only four used an evidence-based program. The City could 
convene providers to discuss these different programs and encourage providers to consider which 
might be appropriate for their settings. 


e Encourage tutoring. Among effective programs, tutoring has the greatest evidence of effectiveness 
and should be particularly encouraged. Philadelphia’s Read by 4 campaign has identified increases 
in tutoring options as a priority. Tutoring programs can be staffed by volunteers with adequate 
support, supervision, and training. In addition, the dosage required to be effective is much lower 
than for group-based programs. However, tutoring programs using volunteers need an 
infrastructure for recruitment, screening, training, and supporting volunteers, and also need access 
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to data for diagnostic and progress monitoring. 


e Allow for variation in programs and practices adopted. As the City considers strategies for the 
first phase of the OST strategic plan, it should allow for variation in programs and practices that fit 
varied OST settings. Interviewed OST providers were clear that a “one size fits all” program would 
not work. In addition, providers want to retain the best of OST and not be forced to give up other 
valuable aspects of their program. Among the evidence-based programs were a range of 
approaches, some of which would require less adaptation for OST settings than others. For 
example, computer-based programs or an afterschool enrichment program like KidzLit are among 
the most affordable and do not require certified staff. While they address a limited range of 
outcomes (phonics for computer-based programs and reading attitudes for afterschool 
enrichment), they may offer a “first tier” of OST literacy programming for providers that don’t have 
the resources to offer other types of OST literacy programming. 


e Foster other practices to support literacy which draw on strengths of OST programs. 
Providers that are not able to offer evidence-based programs may be able to support early literacy 
in other ways. For example, even though OST programs reported that it was challenging to find time 
for parent engagement, OST programs are well positioned to engage families, a critical support for 
early literacy. OST providers may interact more frequently with parents than schools because they 
interact with parents when they are dropping off or picking up their children. In addition, they are 
often staffed by community members who may already have relationships with parents.#4 If OST 
providers prioritize parent engagement and receive support for it, they can help parents 
understand the importance of supporting early literacy and suggest strategies to bolster it. One 
early literacy leader suggested that OST providers could help parents understand their child’s 
reading level and where to find it on report cards. In addition, OST providers are able to offer 
activities that are engaging and can help develop a love of reading and writing. 


2. Encourage collection and use of data. 

e Support OST providers with data collection and outcomes assessment. Access to data for 
diagnostic and progress monitoring purposes is key to providing targeted literacy supports. When 
providers had access to school data, it was primarily through report cards. While report cards area 
useful start and often include a student’s reading level, it can be challenging for providers to get 
access to report card data for all students via requesting it from parents. Schools have access to this 
data as well as more granular data that an OST provider could use to better target student needs. 
But accessing this data was one of the top challenges reported by providers, likely complicated by 
confidentiality issues. The City and the school district could explore whether it would be possible to 
create an agreement to allow schools to share student data with OST programs under certain 
conditions. In addition, Read by 4» and POSTLI could support providers to learn how to use reading 
levels in their planning and in conversations with parents to engage them in OST and other literacy 
efforts. 


e Provide guidance for OST providers regarding the early literacy skills that most need 
targeting for different grade levels and subgroups of students. Different evidence-based 


34 RFA’s Scanning the System report found that Philadelphia’s OST providers consider it somewhat or very important to hire staff who reside in the 
community of the program. 
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programs were shown to be impactful for different age groups and skill areas. As providers choose 
programs that best fit their context, they need guidance from the School District and other early 
literacy experts in Philadelphia regarding the skill areas they should target. While balanced literacy 
instruction is recommended during the school day, OST providers may be more able to target 
specific components of literacy that are a need for the students they serve (e.g., data might show 
that a particular student or group of students have strengths in phonics but need more support in 
reading comprehension). 


3. Fill gaps in training and literacy expertise. 

e Expand literacy-focused training opportunities. A majority of OST providers offered pre-service 
training to frontline literacy staff but they did not offer enough of it. In addition, while providers 
offered ongoing support and coaching, this was not often formalized or offered by someone with 
literacy expertise. Providers reported barriers to training including the cost and the availability of 
part-time staff. The OST system could address this gap, as well as the gaps created by staff turnover, 
by offering additional training resources at key times during the year -particularly prior to the start 
of the school year and summer programming. These should be offered in a variety of formats to 
accommodate the schedules of part-time staff. In addition, interviewed providers reported wanting 
more opportunities to share promising practices with each other. The City OST system could 
organize events that bring providers together to share promising practices with each other. 


e Provide training and other supports for OST providers with regard to ELL students. The vast 
majority of OST providers reported serving at least a few ELL students. At the same time, only 
about one-third of providers offered training focused on serving ELL students and had books 
available in other languages and about other cultures. One provider recommended that the City 
convene providers by regions of the City to share promising practices around working with ELL 
students. The OST system could also sponsor centralized trainings on this topic and identify 
recommended materials and resources to work with ELL students. Providers also reported that 
engaging parents for whom English was not their first language could be challenging. One provider 
hired translators for parent events but, again, described this as a potentially costly endeavor. The 
City should consider ways in which it could support providers in engaging ELL parents. In addition, 
the OST system could consider additional per child funding for programs according to the 
percentage of participants or parents who are ELLs in order to finance appropriate materials and 
translation. 


e Provide access to literacy expertise for program development, training, and coaching. Few 
providers had literacy expertise on staff. More providers accessed this expertise through POSTLI. At 
the same time, half of providers did not have access to literacy expertise at all (on staff or through 
external partnerships). The City could look for more ways to increase access to this expertise 
through existing institutions and programs which have in-house literacy expertise such as POSTLI 
or Read by 4, the School District, the Free Library, or local colleges and universities. In addition, 
OST providers should connect with the reading coaches at students’ elementary schools to explore 
how best to enhance the literacy supports available in their schools. 


4. Engage Key Partners: Universities and Funders 
e Engage colleges and universities as sources of literacy expertise as well as volunteers. Almost 
half of providers reported using volunteers to deliver literacy instruction. Two interviewed 
providers reported that many of these volunteers came from colleges and universities. This 
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parallels a finding from RFA’s Scanning the System report, which showed that a large percentage of 
OST programs utilize volunteers, particularly from colleges and universities, which abound in the 
Philadelphia region. Such volunteers could be an asset to the OST system’s early literacy efforts, 
especially serving as tutors, given the effectiveness of volunteer-staffed one-on-one tutoring 
programs. While there are documented challenges in working with college volunteers -they are 
often “episodic and occasional volunteers”?5- formal partnerships with colleges and universities 
could help to establish and support more stable volunteer pipelines. For example, undergraduate or 
graduate students who are studying education or early literacy may have an inherent interest in 
longer-term involvement with an OST literacy program, and may be able to receive credit for 
course-related volunteer work. In addition, education majors may already have training or such 
training could be incorporated into their classes. Formal partnerships to engage student volunteers 
could also facilitate engagement of faculty who could support trainings and provide other guidance 
at system- or program-levels. One provider also highlighted the use of work-study students which 
could also be explored at a system-level. Many local colleges and universities already have 
programs placing work study students as tutors in school-day and OST programs; often these 
students work in placements for a full year. OST programs could seek to get on the approved work- 
study list. One role for the City could be to develop a directory of college and university contacts 
and programs, particularly in education, that could provide connections to volunteers. 


Engage funders in considering system-wide strategies for supporting OST literacy programs. 
OST programs require adequate funding to operate impactful literacy programs, particularly for the 
staffing and material needs of these programs. At the same time, OST programs operate ina 
resource-constrained environment with funding sources that are fragmented, focused on differing 
goals, and are often non-renewable. In addition, the availability of OST funding often fluctuates with 
changing economic and political realities such as new local, state, or federal administrations whose 
support for OST programming may vary.?¢ Funders could consider engaging in advocacy to increase 
and/or stabilize public funding for these programs. 


It is also important to identify cost savings to help OST providers offer high-quality literacy 
programming within their budgets. Providers reported the biggest driver of cost to be staffing, 
particularly if programs use certified teachers. But, programs using paraprofessionals also have 
additional costs related to the need for a literacy expert who can prepare lessons, train, and provide 
ongoing support for staff. One strategy for cost savings may be to create or expand system-wide 
resources to support paraprofessional staff through POSTLI, Read by 44, or other entities that can 
provide literacy expertise. 


B. Recommendations for Future Research 

This scan of OST literacy practices, along with RFA’s July literature review, illuminated some aspects of OST 
literacy programming, but this research has limitations and important gaps that should be addressed by 
future research. These include: 


The efficacy of non-traditional programs for OST literacy support: The review revealed a lack 
of research on less traditional types of literacy programming such as “infusing literacy” in other 
groups of activities (e.g., drama, arts, sports, board games). Almost all of surveyed OST providers 


35 Cindy Kowal, “Using College Students as Mentors and Tutors,” (2007). 
36 Cheryl Hayes, Christianne Lind, Jean Baldwin Grossman, Nichole Stewart, Sharon Deich, Andrew Gersick, Jennifer McMaken, and Margo Campbell. 
"Investments in building citywide out-of-school-time systems: A six-city study." Philadelphia: Public/Private Ventures & The Finance Project (2009). 
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reported offering “literacy-infused” as well as literacy-focused activities. Future research could 
focus on these literacy-infused programs, which are well-suited to OST programs, to determine if 
and when they are effective in supporting young children’s literacy development and examine best 
practices for this approach. In addition, while some computer-based programs were found to be 
effective, little research was available regarding their implementation. Research should examine 
the use of these computer-based programs in Philadelphia to identify promising practices for 
effective implementation. 


The role of drop-in programs in supporting early literacy: Evidence-based programs all require 
regular participation that reaches particular dosage thresholds. These thresholds are difficult for 
drop-in programs such as libraries, Boys & Girls Clubs, YMCAs, and recreation programs that have 
not typically required youth to commit to programming after school. Yet, these types of programs 
are a key component of Philadelphia’s OST system. Research should examine the role that they can 
play and the types of literacy practices and targeted outcomes that would be both appropriate and 
effective if offered in a drop-in setting. 


The most effective strategies for supporting early literacy of ELL students in OST settings: 
Little research exists on promising practices for supporting ELL students in OST. However, most 
OST providers responding to our survey were serving ELL students. More research is needed on 
programs and practices OST providers use to identify those that are effective. 


The relationship between Key Ingredients and outcomes: The literature review extracted 
common practices from the evidence-based OST literacy programs. However, there is limited 
research that directly ties particular practices to outcomes. Future research should examine the 
relationship between these Key Ingredients of OST literacy interventions and outcomes. 


The evidence base for afterschool enrichment programs and programs using 
paraprofessional staff: In RFA’s literature review, the evidence base for afterschool enrichment 
programs was the weakest of all program types. Therefore, it’s important to further study programs 
that show promise in this setting. YET is a local afterschool enrichment program currently 
implemented by 14 providers in Philadelphia. It was rated in RFA’s literature review as in need of 
more research because rigorous research comparing YET student outcomes to a comparison group 
had not been conducted. YET deserves further research because it is a balanced literacy program, 
i.e., addresses all five skills areas identified by the NRP, and few evidence-based OST programs 
adopted a balanced literacy approach. A second program that deserves further research is KidzLit. 
KidzLit is a curriculum utilized by three providers in Philadelphia. One program, Mercy Housing 
KidzLit, which centered on the KidzLit curriculum, had moderate evidence of its effectiveness for 
impacting reading attitudes. However, this was based on one study which should be replicated in 
other settings. In addition, KidzLit is tailored to paraprofessional staff and is an affordable program 
which could be particularly applicable to OST. Further research on both YET and KidzLit could be 
helpful in identifying more cost effective approaches and strategies for OST literacy supports in the 
afterschool setting. 


The role of external literacy experts, particularly POSTLI, in boosting the quality of OST 


literacy programs: The Philadelphia Out of School Time Literacy Initiative (POSTLI) plays a 
central and unique role in Philadelphia’s OST literacy landscape, providing literacy expertise to OST 
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programs who do not have such expertise on staff. POSTLI, in existence since 2000, is staffed by 
literacy experts including reading specialists and certified teachers, and provides technical 
assistance, coaching, and training for OST programs seeking to offer literacy support. The evidence- 
based programs in our literature review did not report local partnerships for literacy expertise like 
POSTLI and it is unclear whether an external literacy expert such as POSTLI can play the same role 
or have the same results as internal literacy experts. At the same time, POSTLI could be a cost- 
effective vehicle through which to channel literacy expertise to more OST providers. Research 
should examine the use of POSTLI and other external literacy supports as compared to internal staff 
literacy experts and their outcomes of programs using both types of literacy expertise. 


The quality of implementation: The adoption of effective programs and practices does not ensure 
that these are implemented with fidelity or with all the supports that are needed to be high quality. 
As the OST system moves forward with particular programs and types over the next three years, 
research should assess the quality of implementation. For example, a majority of providers 
reported that their program content aligned to school curriculum or standards. What does this 
mean? How do they make this assessment and what difference does this alignment make for 
programming? 


The role of the OST system in boosting early literacy: As Philadelphia’s OST system undertakes 
a focused effort to engage OST programs to address early literacy, future research should follow 
these efforts and document the extent to which providers engage in the effort, the strategies they 
adopt, the quality of their implementation, and ultimately the outcomes of these efforts. Such 
research could add to the field’s understanding of the role an OST system can play in impacting 
critical needs of children and barriers and supports to these efforts. 


31 


Appendix A. Students Served by Surveyed Providers 
Surveyed providers were asked to approximate how many of their students were ELL students and what 
proportion were struggling readers. Figures Al and A2 show the percentage of providers who reported 
that they served “none”, “a few”, or “about half’ or more of each type of student. 
Figure A1: Proportion of Participants that are ELL Students 
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Figure A2: Proportion of Participants that are Struggling Readers 
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Figure A3 below displays variation across program types in the extent to which literacy supports were 
provided for all students, for only those who were identified as needing it, or if students could choose to 


participate. 
Figure A3: Which Students Receive Literacy Supports in OST Programming? 
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Appendix B. Curriculum Use 


Surveyed providers were asked whether their program uses a literacy curriculum. Table B1 shows the 
percentage of providers who reported that use a packaged curriculum (either an evidenced based on 
studied in the Scanning the System report or another curriculum), a curriculum designed by their own 
organization, or those that do not use a curriculum. 


Table B1: Curriculum Used by Providers 


Evidenced-based 


curriculum 
% # 
Tutoring programs 20% | 3 
Afterschool enrichment 17% 6 
programs 
Summer enrichment 17% 6 
programs 


% of providers using a packaged curriculum (R@UGIE 


% not using a 


Other packaged internally designed Guiviculum 
curriculum curriculum 

% # % # % # 
0% 0 40% 6 40% 6 
5% 2 78% 28 0% 0 
8% 3 75% 27 0% 0 


The table above shows: 


e Few providers used a packaged curriculum; the rest designed their own curriculum. Among 
surveyed providers who designed their own curriculum, few had literacy expertise on staff, despite 
the fact that such expertise would be important for designing an early literacy curriculum. 

e One-quarter or fewer providers used a packaged curriculum. However, most of the providers 
using a packaged curriculum were using one of the evidenced-based curricula.?’ Other curricula 
used by providers included Slingerland, iReady, and the Basic Five for Life Program. 


37 The evidence-based curricula were: Read for Success, Reading Coaches, Youth Education for Tomorrow (YET), Book Buddies, Houghton Mifflin’s 


Summer Success, KidzLit, and Open Court 
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Appendix C. Staff Training Topics Offered 


Providers were also asked about the topics of training they offered. 


Table C1: Training Topics Offered 
Provide any literacy- 


specific training Provide any general Provide 
(e.g., building a library, training Provide training 
supporting ELLs, read (e.g., child development, training to about 
alouds, guided behavior management, SunporEtic iterae 
reading/writing, vocabulary, family engagement, PP - 
phonics, comprehension, student background) games 
fluency, literacy games) 
% # % # % # % # 
; 100% 15 93% | 14 a7 4 87%) 13 
Tutoring programs 
Afterschool 94% 33 91% 32 37% 13 | 80% 28 
enrichment 
programs i 
Summer 97% | 33 91% Ex 29% 10 88% 30 
enrichment 
programs 


The table above shows: 


e The vast majority of respondents (91-100%) reported that the training they offered included 
both literacy-specific and general youth development topics. Most providers also offered 
specific training in literacy games. Interviewees provided examples of these literacy games, 
such as sight word Bingo or Go Fish for word families. 

e Only one-third or fewer OST providers reported offering training for staff on supporting literacy 
for ELL students. One interviewed provider highlighted this as an area in which they would like 
more training from the City OST system and the School District. 


38 The training topic of “literacy games” is included in the column for “literacy-specific training” and also pulled out into its own column since these 
activities are particularly of interest for OST programs. 
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